fix: handle non-ASCII languages in ROUGE-1 eval tokenizer#6483
Open
guptaishaan wants to merge 1 commit into
Open
fix: handle non-ASCII languages in ROUGE-1 eval tokenizer#6483guptaishaan wants to merge 1 commit into
guptaishaan wants to merge 1 commit into
Conversation
The `_calculate_rouge_1_scores` function in `final_response_match_v1.py` uses Signed-off-by: Ishaan <ishaangupta0408@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
_calculate_rouge_1_scoresfunction infinal_response_match_v1.pyusesrouge_scorer.RougeScorerwith its default tokenizer, which strips everycharacter that does not match
[a-z0-9]. For languages such as Thai, Chinese,or Arabic—where words contain no ASCII characters—every token is removed and
the scorer always returns an F-measure of 0.0, causing eval to report a
mismatch even for identical strings.
The fix introduces a
_UnicodeTokenizerthat is used automatically whenevereither the candidate or reference text contains non-ASCII characters. It splits
on whitespace and, for each non-ASCII word, emits individual Unicode characters
as tokens so that ROUGE-1 overlap is computed at the character level—a
well-established fallback for scripts that lack whitespace word delimiters.
ASCII text continues to use the existing stem-based tokenizer unchanged.
Two regression tests are added to
test_final_response_match_v1.py: oneasserting that identical Thai text scores 1.0, and one asserting that entirely
different Thai text scores 0.0.
Fixes #3111.